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INTRODUCTION 

This peeper vocuses on the use of a math skills monitoring system and re- 
ports how information collected during thd typleal Levels of Use Interview pro* 
vided the data for assigning each teacher to a configuration^ or pattern of uses 
of the monitoring system* The monitoring system and the technical procedures 
for deriving the configurations from the interview data are reported. Finally, 
how teachers' configurations relate to student achievement Is reviewed. The 
primary objective of the paper Is to describe In detail the methods used to (1) 
discover how teachers were using the progron and (2) cluster the teachers into 
groups with similar patterns of behaviors. The relationships between teacher 
behaviors and student achievement Illustrate one application of the results of 
these procedures, 

THE INNOVATION: SAM 

Beginning in 1975, a Ski lis Achievement Monitoring System (SAM) was de- 
veloped for the purpose of assessing mathematics skills in elementary and 

Ipart of the research described herein was conducted under contract with 
the National Institute of Education and the Fitchburg Public Schools, Fitchburg, 
Massachusetts. The opinions expressed are those of the authors and do not 
nicassarlly reflect the position or policy of the funding agencleSp and no 
endorsiment by them should be Inferred* 
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secondary schools in a local school district. The current procedure is that at 
each six week interval , the system informs classroom teachers, students and 
their parents what skills each child has learned. 

The SAM was developed by classroom teachers with the assistance of an eval- 
uation specialist and a math specialist. All of the district's teachers* in 
grades 2 through 8* were asked to identify the irost critical math skills for 
their respective grade. After grade level consensus was reached, a representa- 
tive group of teachers^ with the specialists, constructed items asure each 
skill* These Items represent the content of the SAM tests. 

Several considerations affected the district -s development and implementa- 
tion of SAM. Firsts achievement monitoring was assumed to be diagnostic in 
nature, presumed to be a basic part of instruction. Test results would identify 
students who need instruction on specific objectives, that Is^ the results would 
provide data for Individualizing instruction. Secondly, the involvement of 
teachers from the inception of the innovation was important In gaining their 
ownership. This meant that they were engaged in the development of the SAM: 
its content, format and reporting process. Third, teachers were to be given an 
important decision- making role with respect to use of the program. It was felt 
that the system would best serve teachers if they had the freedom to use it in 
their own self- prescribed ways. Use was not prescribed by the district and 
teachers* as a result, used it in a variety of ways. 

A key ingredient in the SAM implementation was the service of a district 
math coordinator who responded to individual teacher needs. For instance* every 
six weeks the math coordinator personally delivered the computer printouts 
(processed by the district) to each teacher in his or her classroom. These 
visits were designed to explain the test results* answer questions and provide 
help. The math coordinator supported teachers as they Implemented the SAM in 



their own personalized waj^, and encourar ■ te the test results 

in conjunction with other available st 

PREVIOUS ICH ON S i 

In 1978s the extent to which the ^ had plemented In classrooms 

was assessed in order to evaluate the i ^ - . m s- \ on the math achievement of 
pupils. The district wished to knc whe erences in student achievement 

might be attributable to differentia; use e SAM, In order to do thiSs the 
district contacted the Concerns-Bas% Adopt jn Project at the Research and 
Development Center for Teacher Education at The University of Texas at Austin, 
This research team had been engaged for several years previously in research on 
change In schools and colleges* The work this project has been doing seeks to 
understand how the adoption of educational Innovations occurs* how the Individ* 
uals involved in implementing change are affected, and how to facilitate this 
Implementation* The research is based on the Concerns-Based Adoption Model 
(CBAM), which describes what happens to individual users as an innovation Is 
implemented* 

In the CBAM, Levels of Use of the Innovation (LoU) (Hall, Loucks, Ruther- 
fords & Newlove, 1975) describes how teachers use new programs. The Levels of 
Use interview (Loucks, Newlove & HalU 1975) provides valuable information for 
assessing and rating the individual teacher's use. Understanding what Is hap« 
pening to the innovation, that is, what teachers are using, is another key di- 
mension of the CBAM work* This is the concept of Innovation Configurations 
(Hall & Loucks, 1978) and Is baseline data collected in the LoU interview. 

The Levels of Use (LoU) interview was utilized to gather data on teachers* 
use of the SAM, This interview provided information on how teachers were using 
SAM, i,e*, at which of the eight Levels of Use; it also provided important in- 
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formation on what variations of the possible components, or conf IgurationSs 
teachers were using. 

District evaluation personnel defined a "user" as a teacher who utilized 
the SAM test In instructional decision-making. A "nonuser" was a person who ad- 
ministered the SAM testSs received the results^ but took no subsequent action 
based on the results. 

Constructing a Configuration Checklist 

In order to describe the teachers* patterns of use, or configurations* 
district staff were asked to respond to three questions* What are the critical 
components of SAM? What might be observed when the SAM Is operational with 
classroom teachers? What would teachers, pupils and/or others be doing? Based 
on responses to these questions and the Interviewers' prior experience with a 
similar math program configuration search, a comprehensive list of all likely 
components and variations was prepared. 

This list was used to structure Interviews with a number of teachers using 
SAM* Information gained in this interview resulted In the elimination of some 
compon-nts and a more orderly and logical sequencing of the remaining compo* 
nents. Appendix A represents this configuration checklist* This checklist was 
used to codify the practices of 71 teachers vA)o described their use of SAM dur- 
ing an LoU interview* 
R esults 

As the taped interviews were being rated, each teacher was assigned to a 
configuration group: "minimal use," "mixed use," and "committed use," These 
classifications were based on a clinical analysis, an overall gestalt formed 
after visiting with teachers and listening to the taped Interviews. 
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The Levels of Use ratings demonstrated that all the teachers did use SAM, 
that iSs every teacher took some action based on the results of the SAM tests. 
Analyses of student achievement data Indicated that increased use of SAM con- 
tributed to student learning (Reidy & Hord, 1979)* A state department team 
validated SAM as a model program and provided funds for a diffusion grant to 
help other schools adopt SAM for their use. 

The Measurement of Configurations In Other Studies 

The CBAM Project has developed the concept of Innovation Configurations to 
describe the various ways that individual teachers operationalize any given 
innovation* The measurement of configurations has varied from one study to 
another, and vt Is of some concern to members of the CBAM Project that readers 
of papers which report configuration data do not become confused by different 
procedures which have been used to describe configurations. In other studies 
which have used the concept of conf igurations, the developer of the innovation 
has been Interviewed, one or more sites visited to see the innovation in prac* 
tice, and a configuration checklist drawn on this basis. The developers of the 
SAM did not specify teacher behaviors, except tb say that they expected the 
teachers to use the printouts to guide their math instruction. Thus, the con- 
figuration checklist developed for this study differs from those In other stud- 
ies because it was developed for an innovation which did not have a great deal 
of developer specification. A discovery approach was used In this study to de- 
fine the components of the Innovation as well as to describe teacher behav- 
iors. 
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THE CURRENT STUDY 



In 1979s P^^t of a second evaluation of the SAM in preparation for 
national validation as an exemplary program, Level of Use interviews were again 
employed to describe the current state of implementation. In addition to LoU 
ratings of each teacher, the interview was again expected to reveal the extent 
to which teachers relied on the SAM to guide their math instruction. Analyses 
of data collected in the interviews would indicate if there were discernable 
patterns of use, or configurations. The sample of teachers was expanded to 96 
and included nine teachers who were new to SAM* 

A "New" Checklist 

The attempt was made In the interview to identify and describe alV prac- 
tices in which the teachers engaged with SAM, The Interview tapes were reviewed 
and all teachers* behaviors were noted. Redundancies were removed and a list of 
14S items were extracted as exemplars of teacher practices in relation to the 
SAM. These Items were edited and collapsed into twenty-nine statements which 
represented the variety of teacher practices or behaviors specified by the 
teachers. The refined configuration checklist contains the 29 items arranged 
into five groupings (see AppendiK B). 

Since the 29-item checklist was not available at the time of the inter- 
views, it was not possible to specifically ask about each of the items when 
talking to the teachers* Even so, it seemed appropriate to make inferences on 
those items which were not specifically addressed in the Interview. When the 
29-item checklists were filled out, raters marked every item as True or False, 
and also Indicated whether data were specifically available to make that rating- 
Making these Inferences was a difficult task; the failure of a teacher to 
mention the presence of a particular practice does not Imply that the teacher 
does not engage in that practice. However, each interviewer had made a rather 
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thorough attempt to get each teacher to describe his or her use of the innova- 
tion in detail* The checklist used in the interview contained the same cate- 
gories of behaviors as the 29-1tem checklist. In every interview^ the teacher 
was asked specific questions about the use of SAM objectives, instructional 
materials, testings printouts and remediation. Most teachers gave very specific 
descriptions of their practices in the classroom. 

Reliability of the Ratings 

Six interviews were independently rated by a second person to assess the 
reliability of the ratings. On these tapes, there was B7% agreement on the 
True/False rating for items on which one or both raters indicated an inference 
was necessary, 94% agreement was found on items both raters agreed no inference 
was necessary. These statistics convinced us that very good reliability of 

ratings were being maintained even under "high inference" conditions, 

i» 

Analysis of Configuration Checklist Data 

Using ratings on each of the 29 items, a value of one (1) was assigned for 
items marked "true" and a value of two (2) for items marked "false," Teachers 
then were clustered Into groups which had similar practices* In order to ensure 
that the groupings were related to the extent of implementation, each teacher 
was assigned to a high, rediums or low Implementation group by the person who 
had persDnally interviewed that teacner. These assignments were used as start- 
ing points In a non-hierarchical cluster analysis* The procedure used was 
developed by Forgey (1965), and programmed by Anderberg (1973)* Brief ly» the 
procedure consists of the following sequence of steps: 

1* Begin with any desired initial clusters of the data. 

2. Compute the centroid of each cluster (in this case the mean 
of each item on the checklist). 
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3. Considering each teacher's data, assign that teacher to the 
cluster which has the nearest centroid, 

4. Alternate steps 2 and 3 until no teachers change their cluster 
memberships. 

With the data on the 29-1tem checklist and the Initial clusters given by 
the interviewers' clinical ratings, only two iterations were required to reach a 
stable clustering. At that point, each teacher was assigned to the group clos- 
est to his or her "profile" on the checklist. Most of the changes in group 
membership were shifts from the medium to high and redium to low groups. Ini- 
tially, sixteen teachers had been assigned to the low implementation group, 
twenty were assigned there after the cluster analysis. Also, 40 teachers ended 
up in the high group while 29 had originally been assigned there. It seems that 
the raters were hesitant to assign teachers to groups outside the medium clus- 
ter, but not surprised at the results, and were in some cases pleased that "the 
computer" had put the teachers in the final groupings. 

In reviewing the individual teachers who had been moved from one group to 
another, the raters were in agreiment with the new cluster assignments, with one 
exception* One teacher, placed in the lew group through clinical assessment, 
moved to the high group as a result of computer analysis. This teacher had 
arranged a sophisticated and individualized procedure for student remediation; 
however, no class time was provided for students to engage in the designed 
activities. In fact, they were denied "time on this task" by the requirements 
of the teacher's ongoing math program which did not integrate SAM components. 
This unique configuration really did not fit any cluster well (see p. 16)* 

Figure 1 shows the proportions of teachers in each cluster who were rated 
as true for each statement on the checklist. Major differences between clusters 
were observed on items 1 and 2 under Instructional Materials, items 1 and 4 
under Testing, item 2 under Printouts, and items 1, 2, 3, 4* S, 6, 8, 9, and 11 



Figure 1 
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Results of fliirarchical Cluster Analysis 
on Fitchburg Confiauration Checklist 

% True 



Hi 


Med 


Low 












Objectives 


70 


12 


5 


1 


Uses the SAM objectives as the primary curriculum guide for 
ongoing instruction. 


98 


95 


80 


2 


Devotes class time to teaching some or all SAM objectives 
for ongoing math Instruction not remediation. 


4 


3 


20 


3 


Increases attention to teaching SAI^I objectives Inmediately 
previous to SAM testing (1 .e. , "preps" for SAM tests). 


78 


95 


95 


4 


Teaches math objectives other than SAM objectives (whether 
or not SAM objectives are taught). 


33 


32 


20 


5 


Instructs resource room students 1n SAM objectives. 








Instructional Materials 


65 


20 


10 


1 


Uses pre-packaged materials (IMP or similar) keyed to SAM 
objectives for ongoing math Instruction. 


78 


17 


15 


2 


Uses personally grouped materials keyed to SAM objectives 
for ongoing math Instruction. 








Testing 


93 


20 


15 


1 


Administers tests specifically focused on SAM objectives be- 
tween SAM tests. 


98 


97 


90 


2 


Administers other math tests besides SAM tests. 


5 


5 


0 


3 


Administers more than one level of the SAM to individual 
students (I.e., a student takes two SAM tests simultaneously 


90 


69 


40 


4 


Moves students from one level of the SAM to another during 
the year. 








Printouts 


TOO 


92 


95 


1 


Sees that students each receive a copy of each SAM printout. 



TOO 83 30 2 Provides that each student has a readily accessible record of 

perforinance (printout or chart) on previous SAM tests. 
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Figure 1 (contlnuid) 



% True 
Med Low 



83 80 3 Expects students to take each (every) printout to parents, 

9i 90 4 Sandis printouts homa with students at end of year (rtgard- 
less of other tlmesK 

3 0 5 Requests parents to sign to show they have received printouts 

Z6 5 6 Shares SAM printouts with child's other teachers (i.g.j Title 
I). ^ 

6 0 7 Posts SAM printouts In classroom (e,g., on the wall). 

Remediation 

98 25 1 Requires students to show mastery or work toward mastery of 
objectives missed on SAI^ tests, 

49 15 2 Creates problems or exercises "on the spot" to reteach missed 
objectives, 

37 10 3 Uses pre-packaged materials keys to SAM objectives (IMP or 
similar) for remediation work, 

37 5 4 Uses personally grouped materials keyed to SAM objectives 
for remediation. 

63 35 5 Reviews SAM results with class as a whole within a few days 
of their return, 

37 15 6 Reviews SAM results with individual students within a few 
days of their return, 

37 40 7 Focuses remediation on whole class (based on SAM results). 

63 20 8 Forms small groups based on SAM results for the purpose of 
remediation, 

66 5 9 Focuses remediation on Individual students (based on SAM 

results), 

100 60 10 Compares previous results on SAM test with current results, 

100 30 11 Assists students to be aware of progress made since last test 
(or over the year) . 
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under Remediation, Generally, the high Implementation group contained a 
majority of teachers rated as performing the practice, while a minority of 
teachers in the medium or low group exhibited the practice. One interesting 
exception to this 1s the distribution of true ratings for item 6 under Remedia- 
tion. Twenty percent of the high group review the results of the test with 
the class as a whole^ sixty-three percent of the medium group do this, and 
thirty-five percent of the low. It appears that teachers In the high group re- 
view the results with students on an individual basis, teachers In the Interme- 
diate group tend to review the results with the class as a whole, while those In 
the low group do not go over them with students at all. 

Relationship of Implementation to Student Achievenfent 

In order to assess the relationship of the teacher's innovation configura- 
tion to student achievement, the following analyses were done, SAM test data 
were available on 2911 students for the 1978-79 academic year* Each student 
should have taken six SAM tests during this period, at approximately six week 
Intervals* There are seven levels of the SAM tests for grades two through 
eight. At each level, six forms of the test exist. Students receive a differ- 
ent form of the test each time they are tested at a given level. 

Because students frequently move from one level to the next during the 
year, it was necessary to construct a common scale onto wlilch scores on every 
test could be projected. This scale was constructed by administering more than 
one level of the test to students at the same time and comparing performance on 
the two tests, A total of 498 paired tests were administered, distributed 
across the levels. The common scale was constructed by examining the mean 
scores on adjacent level tests taken by the same students, and adjusting the 
scores on each test so that the mean common scale score was the same regardless 



12 

of the level of the test used to, compute it. Although it was recognized that 
this procedure was less than ideal, the use of the common scale scores as a co- 
variate as well as the dependent variable mitigates its weaknesses. 

An analysis of covariance was performed using the final SAM test of the 
year as the dependent variable, the three implementation groups as the indepen- 
dent variable, and the initial SAM test, grade level and sex as the covariates. 
If a student had missed the first SAM test but taken the second, the second 
score was used in its place (N=145). If a student had missed the final (sixth) 
SAM test, but had taken the fifth, the score on the fifth was used in place of 
the final score (N«85). A total of 2299 students had both initial and final 
scores following this procedure. 

The students' test data was aggregated by sex and grade level, resulting i 
160 data points, representing 80 teachers' classrooms. Each of these teachers 
had been assigned to one of the implementation groups at the end of the year. 
There were 37 teachers in group 1 (high implementation), 30 teachers in group 2 
(moderate implementation), and 13 teachers in group 3 (low Implementation). 
Sixteen teachers, predominately in the low implementation group, were not in- 
cluded because none of their students had received SAM tests in the fall. 

Sex and grade level were not significantly related to achievement on the 
final SAM test when the Initial test was used as a covariate. However, there 
was a significant Interaction between configuration groups (F = 7.17, d.f . ^ 
2,154, p = .001). 

The nature of this Interaction is shown in Figure 2. It is readily appar- 
ent from Figure 2 that classrooms with teachers In the high and moderate imple- 
mentation groups performed better than classrooms with teachers in the low 
implementation group at higher levels of the Fall 78 scores. It was surprising 
to see that the moderate implemenation group did better than the high implemen- 
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Figure 2 

Relationship of Degree of Implementation and 
leveinent of the SAM Tests 
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tat ion group, A classroom with a mean score of 40 in the fall has an expected 
mean score 1n the spring of 41*1 (low group), 45.15 (high group), and 47.5 
(moderate group)* A test was performed to determine whether the high and moder- 
ate groups could be represented by lines having the same slope; a marginally 
significant difference was found (F ^ 3*75, 'd,f.^ 1,155* p ^ .054), indicat- 
ing a difference In achievement does exist between the moderate and high imple- 
mentation groups* 

Thus, it appears that teachers using the SAM program in a moderate to high 
implementation mode have students who perform better on the SAM tests, but only 
in classrooms with higher fall scores. These differences completely disappear 
at low levels of the tests. Because scores on the SAM are closely related to 
grade level (r ^ .94), it can be argued that the differences in achievement are 
more pronounced at higher grade levels. 

It was somewhat surprising that the rroderate implementation group had high* 
er achievement than the high implementation group; several additional analyses 
were run in an effort to understand this* First, analyses of covarlance were 
run using each of the 29 items on the configuration checklist as the independent 
variable, the posttest dependent variable, and the pretest as the covarlate. 
The results of these tests are presented in Table 1* 



Table 1 

Items on the configuration checklist significantly related to achievement 
using an analysis of covarlance test 



Item 








F 




direction 


17. 


Shares SAM with 


other teachers 


4 


.52 


.04 


true/false 


23. 


reviews results 


with whole class 


7 


.58 


.01 


true/false 


24. 


reviews results 


with individual students 


4 


.93 


.03 


false/true 
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Only three Items were significantly related to the achievement of stu- 
dents according to this analysis* One of these was in a direction opposite that 
expected teachers who review the results of the SAM with individual students 
have lower achlevenient than those who do not do so* 

A second analysis was nrare informative a discriminant analysis using the 
29 items on the checklist to separate the groupSs followed by a plot of the 
group membership on the two dimensions resulting from this analysis* The SPSS 
subprogram discriminant was used to select the first 15 items which maximally 
discriminated the three groups and to do the plot. The number 15 was choseni 
somewhat arbitrarily^ because this was 50^ of the 29 available items* Figure 3 
shows the groups along the two discriminant functions found using this approach. 
It seems that the first function separates the groups along a dimension from 
high Tmplementat ion to low (left to right on the horizontal axis), while the 
second function separates the groups along some other dimension. Since group 
2 had the highest achievement, and group 3 the lowest, the vertical axis was 
tentatively called the "achievement" dimensionSp high achievement being toward 
the bottom of the plot. 

It is noteworthy that the one teacher moved from group 3 (low implementa* 
tion) to group 1 (high Implementation) by the cluster analysis is represented on 
the plot by the lone "1" In the upper center of the plot* This teacher*s mini- 
mal use of a sophisticated system geared toward seems to constitute a unique 
configuration. It just happens to be slightly closer to group 1 than 2 or 3. 

The steps In the discriminant analysis, and the final coefficients for the 
IS variables entered into the analysis are shown in Table 2* These numbers in- 
dicate that these 15 items are the most effective in separating the three 
groups. The previous use of cluster analysis on these same teachers using the 
same data points made it possible to separate these groups much more cleanly 
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Figure 3 

Plot of Discriminant Score 1 (Horizontal) vs. Oiscriminant 

Score 2 (Vertical ) 
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Table 2 

Order of entry and final weights for IS items in the 
discriminant analysis. 



Standardized Discriminant 
Function Weights 



Item 


F ratio 


Function 1 


Function 2 




19 


85.26 


-1.21 


-1.33 


mastery on missed objectives 


8 


25.44 


- .43 


.41 


SAM tests between SAM tests 


24 


12.09 


- .40 


.48 


reviews test with individuals 


1 


9.84 


- .35 


.45 


SAM objectives primary math 


21 


8.03 


- .35 


.25 


guide 
IMP for remediation 


25 


4.07 


.32 


- .23 


remed focused on whole class 


28 


4.61 


- .65 


- .30 


compares previous SAM to 










current 


7 


2.86 


- .51 


.44 


personally grouped materials 










for instruction 


18 


2.62 


.26 


- .15 


posts SAM In classroom 


13 


2.53 


- .47 


- .66 


provides readily accessible 










records 


12 


5.40 


.04 


.60 


each student receives every 










printout 


29 


3.56 


.35 


.56 


assists students to be aware 










of progress 


15 


1.81 


.27 


- .13 


sends printouts home at end 










of year 


16 


1.81 


.26 


- .15 


parents sign for printouts 


5 


U76 


.23 


.16 


insures that resource stu- 



dents get SAM 



is 
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than might be encountered in other studies* Indeed, as can be seen from the 
plot in Figure 3* only four cases might be incorrectly assigned the "wrong" 
groups using these two functions there is a Slight overlap between groups 1 
and 2. 

Looking at the discriminant function weightSs we see that requiring mastery 
on missed objectives is the most indicative item for both the implementation and 
the achievement dimension. The next four items are indicative of high imple- 
mentation but low achievement: administering tests keyed to SAM objectives be- 
tween SAM tests given by the district, reviewing the results of the SAM tests 
with individual students, using the SAM objectives as the primary guide for on 
going math Instruct ion^ and using IMP or other pre-packaged math materials for 
remediation. Focusing remediation on the whole class is Indicative of low 1m* 
plementation but high achievement^ and so on. The negative weights on coeffi- 
cients which are positively related to either dimension are due to the polarity 
of the discriminant functions: high implementation is toward the left (nega* 
tive) and high achievement toward the lower (negative) portions of the plot* 

In order to Investigate the relationship of achievement to the discriminant 
functions, the teachers' discriminant scores were used to predict achievemant. 
The procedure used was to predict posttest performance using the pretest per- 
formance^ and then see if the addition of the first or second discriminant 
scores significantly increased the accuracy of prediction. This test proved 
non-significant for the first discriminant function (F ^ ,003, p - ,95)* but 
significant for the second function (F = 5.19, p ^ .02). Thus, it seems that 
the two discriminant functions were correctly Interpreted, However, the ques- 
tion remains concerning what it is about teachers with nwderate implementation 
scores that results in high achievement gains. 
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Interpretation of These Analj^ses 

Teachers who limited their math instruction to the SAM objectivis did not 
have the highest achievenient on the SAM tests. Apparently, the most effective 
teachers used the SAM tests as an indicator and a check on the effectiveness of 
their instruction. It should be noted that the local developers of the SAM 
tests did not intend for these objectives to be the only math content. Perhaps 
the term "high implementation" was incorrectly applied to those teachers who did 
not modify or deviate from the program. These teachers were, in a very real 
sense, not implementing the program in the manner in which it was intended. The 
teachers who modified and added to the program were actually using it in the 
manner envisioned by the developers. 

Even with these considerations in mind, some of the results are surprising. 
How is it that teachers who go over the SAM results with Individual students 
have lower achievement than those who do not? Why is the use of pre-packaged ' 
materials for remedial work apparently less effective than not using these 
materials? When one considers that it is performance on the SAM tests which was 
used to measure achievement In this study, how can it be that administering 
tests focused on SAM objectives to students between the district's SAM tests is 
negative ly related to achievement gains? 

One possibility may be the manner In which the scores on the different 
levels of the SAM were translated into common scale scores (See p. 12). Stu- 
dents on the higher level tests may not be given enough "credit" after moving 
from one level to the next. This and other possibilities will be explored in 
the future study of this innovation and its implementation. 
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IMPLICATIONS 

The major significance of this work is that it demonstrates that the be- 
haviors of teachers using an innovation can be defined by Interviewing teachers. 
The developers of the innovation did not formany define the innovation. They 
facilitated group Tneetings in which the teachers defined the content of the 
tests, provided some support structures, and allowed the teachers to engage In 
whatever behaviors seemed appropriate. One outcome of this approach is that all 
teachers are "users" of the innovation, because no specific behaviors are Iden- 
tified as necessary for "use" of the innovation. This paper illustrates that 
teacher behaviors can be documented quantitatively based on interviews with 
teachers. 

For Researchers 

Researchers might be interested in both the procedures and results of this 
study. Studies of Innovation implementation have repeatedly showi that teachers 
do not all comply with the demands of even the most structured Innovations. The 
attempt to create "teacher proof" curriculum packages has not succeeded in 
eliminating variation in use of innovations. In this study we were able to 
identify quite specifically those variations in the use of the SAM and those 
that were positively related to student achievement. 

For Staff Developers 

Information provided by the configuration checklist could be of great help 

in staff development efforts. Knowing that certain teachers are or are not en- ' 
gaged in certain practices should enable persons responsible for workshops to do 
a more effective job. In addition, staff developers can utilize this procedure 
in order to determine what is actually taking place in the classroom, indepen- 
dent of what the developer expects. The practices of some teachers can also be 
communicated directly to other teachers. One of the outcomes of the reported 



work on thi SAM was to send all the teachers interviewed a list of activities 
extracted from the interviews. Each of these rethods of using the program had 
been "field tasted" by other teachers in the district. 

For Developers 

Finany^ the developers of the innovation were able to see which practices 
had gained widespread use and which seemed to be most effective. Some compo- 
nents of the innovation were either more difficult to implement than others or 
less acceptable to the teachers. Program developers who are considering curri- 
culum improvement efforts might use this discovery approach to configuration 
checklist development to identify the practices which are actually occurring be* 
fore engaging in efforts to define new practices. It may be possible that what 
Is necessary is a dissemination effort in which teachers share what they are 
dolngp rather than undertaking a new start. 
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Appendix A 
1978 Configuration Checklist 



School Name Grade 

Please check any items which describe this teacher *s use. A star will indicate 
emphasis by the teacher. 

1, Administers tests: 

SAM test to math class students 

^ Teacher made tests 

Commercial 

Other 



2, Uses objectives: 

SAM objectives 
Textbook objectives 

Teacher-generated objectives 

Other 



3. Groups for concept development/skills instruction: 
Large group , heterogeneous 
Large groups homogeneous 
Small stable groups (little movement) 
Small changing groups (frequent movement) 
Individuals 
Other 



4. Uses test results for instruction: 
Regroup classes 

Regroup kids (within classroom) 
____ Plan work/activity for Individuals 
Plan work/activity for groups 
Plan work/activity for total class 
Other 

erJc 



Additional uses of test results: 
Share with parents by sending a copy home 

Share with parents In conference 

Place in child's folder 
___ Post on wall 

Other 



Moves individuals to another level of test: 

Teacher judgement Teacher tests 

Dally class work SAM test results: 

Other ^ percentage of items passed 

percentage varies with pupils 



Uses Instructional resources: 
IMP 

Textbook 
Workbook 
Other 



Games 

Manipulati ves 

Teacher made worksheets 

ConiTiercial worksheets 



Influences report card grades: 

SAM test results considered 
SAM test results not considered 
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Appendix B 

The 29-Item Configuration Checklist for SAM Math 

Objectives 

1 Uies the objectives as the primary curriculura guide for math, 

2 Devotee class time to teaching some or all objectives specif ically* 

3 Increases attention to teaching objectives inmiediately previous to 
SAM testing (i.e., "preps" for SAM tests), 

4 Teaches math objectives other than SAM objectives (whether or not SAM 
objectives are taught). 

5 Instructs resource room students in SAM objectives. 
Instructional Materials 

1 Uses pre-packaged materials (lOT or similar) keyed to SAK objectives 
for ongoing math instruction* 

2 Uses personally grouped materials keyed to SAM objectives for ongoing 
math Instruction, 

Testing 

1 Administers teats specifically focused on SAM objectives between teste. 

2 Administers other math tests besides SAM tests. 

3 Administers more than one level of the SAM to individual students (i.e., 
a student takes two SAM tests simultaneously)* 

4 Moves students from one level of the to another during the year* 
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Printouts 

1 Sees that students each receive a copy of each S^ printout. 

2 Provides that each student has a readily accessible record of performance 
(printout or chart) on previous SAM tests, 

3 Expects students to take each (every) printout to ^.arents* 

4 Sends printouts home with students at end of year (regardless of other 
times) , 

5 Requests parents to sign to show they have received printouts, 

6 Shares SAM printouts with child's other teachers (e.g.* Title I). 

7 Posts SAM printouts in the classroom (e.g., on the wall). 
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Remediation 

1 Requires students to show mastery or work toward mastery of objectives 
missed on SAM tests. 

2 Creates problems or exercises ''on the spot" to reteach missed objectives. 

3 Uses pre-packaged materials keyed to S^I objectives (IKP or similar) for 
remediation work. 

4 Uses personally grouped materials keyed to SAM objectives for remediation, 

5 Reviews SAM results with class as a whole within a few days of their 
return. 

6 Reviews results with individual students within a few days of their 
return* 

7 Focuses remediation on whole class (based on SAM results) * 

8 Forms small groups based on results for the purpose of remediation. 

9 Focuses remediation on individual students (based on SAM results) • 

10 Compares previous results on BMi test with current results. 

11 Assists students to be aware of progress made since last test (or over 
the year) * 
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